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Abstract 



Successful organizations, both public and private, rely heavily on 
"word-of-mouth recommendations" for their products and services. 
Financial viability of any organization depends profoundly on formal 
and informal networks of customers who pass their assessments of an 
organization's performance to acquaintances, friends and families. This 
analysis attempts to explain students' recommendations to attend 
NIACC to families, friends and acquaintances. Information to help 
answer this question comes from the Community College Survey of 
Student Engagement (CCSSE). This analysis explores the relationship 
between students' responses to the CCSSE question, "Would you 
recommend this college to a friend or family member," with other 
survey questions dealing with the quality of relationships with other 
students, faculty and administrative personnel. In addition, the 
analysis examines the relationship between student recommendations 
and how they evaluate their "entire educational experience" at NIACC. 
Logistic and multiple regression models identify important predictors of 
student recommendations and their response to the question, "How 
would you evaluate your entire experience at this college?" An 
increase in one unit on the Entire Experience scale increases the odds 
of making a positive recommendation by nearly 13 times, holding the 
environmental variable, student relationships with each other, 
constant. "Quality relationships with faculty" has the most important 
impact on students' entire experience responses. The key role of 
faculty for creating a positive experience at NIACC is highlighted by 
this finding. 
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Executive Summary 



As "enrollment management" becomes more important for the fiscal 
heath of any higher education institution it is important to understand 
the dynamics and importance of 'word-of-mouth recommendations' 
passed on by existing students to their friends and family members. 
What determines students' positive or negative recommendations? 

One source of information to help answer this question is the 
Community College Survey of Student Engagement (CCSSE). The first 
stage of this analysis explores the relationship between students' 
responses to the CCSSE question, "Would you recommend this college 
to a friend or family member?," with other survey questions dealing 
with the quality of relationships with other students, faculty and 
administrative personnel. Included in this stage of the analysis is an 
examination of the relationship between student recommendations and 
how they evaluate their "entire educational experience" at NIACC. In 
the final stage of the analysis the "entire experience" response was 
transformed from an explanatory variable utilized to understand 
student recommendations to a dependent variable to be explained by 
student responses to other CCSSE questions. 

Findings 

Stage One Findings. In the first stage of the analysis logistics 
regression was utilized to assess the relative importance of variables 
dealing with the quality of student relationships with people, 
specifically administrative personnel, faculty and other students. Major 
findings for this first stage of the analysis include: 

■ Positive experiences (measured by a one unit change in student 
response) with administrative personnel, holding all other 
variables constant, improves the student's odds for providing a 
positive recommendation by 54%. 

■ Positive experiences with other students, holding all other 
variables constant, improve the student's odds for providing a 
positive recommendation by 49%. 

■ Positive experiences with faculty, holding all other variables 
constant, improve the student's odds for providing a positive 
recommendation by 21%. 
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There are many other CCSSE survey questions that add to our 
understanding and ability to predict student recommendations. In 
particular the CCSSE survey asks the question, "How would you 
evaluate your entire experience at this college?" If we add this 
response to the model will it improve our ability to understand and 
predict student recommendations? 

Major findings associated with adding the "Entire Experience" response 
to the model include the following: 

■ Positive experiences with faculty and administrative personnel no 
longer are statistically significant and were therefore dropped 
from the explanatory model. 

■ Students' entire experiences at NIACC have a tremendous 
impact on their odds of making a positive recommendation for 
the college to family and friends. 

> An increase in one unit on the Entire Experience scale 
increases the odds of making a positive recommendation 
by nearly 13 times, holding the environmental variable, 
student relationships with each other, constant. 

■ Quality relationships with other students still demonstrate 
significant effects on "Recommendation". A one unit increase in 
student responses to relationships with other students increases 
the odds for a positive recommendation by 51%. 

■ Clearly both variables - students' "entire experiences" at NIACC 
and their "relationship with each other" - are important but 
students' total experiences at NIACC overwhelm all other 
variables in this analysis. 

Stage Two Findings. As students' entire experiences are so critical for 
determining their recommendations to attend NIACC a second stage 
analysis was undertaken to determine factors that account for the 
variance in the "Entire Experience" variable. In other words, "Entire 
Experience" was moved from an independent variable to a dependent 
variable to be explained. 

Student responses on the CCSSE survey provide insights into many 
variables that could possibly impact students' responses to the "Entire 
Experience" question. Among the variables that were of immediate 
interest included the following: 
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Variables for the 
“Entire Experience” Anal ysis 


Name 


Description 


ENVSTU 


Quality of relationships with other students 


ENVFAC 


Quality of relationships with faculty 


ENVADM 


Quality of relationships with Administrative 
personnel and offices 


GPA 


College GPA 


SATACAD 


Satisfaction: academic advising/ planning 


SATCACOU 


Satisfaction: career counseling 


SATCHLD 


Satisfaction: child care 


SATCOMLB 


Satisfaction: computer lab 


SATDISAB 


Satisfaction: disability services 


SATFAADV 


Satisfaction: financial aid advising 


SATJOBPL 


Satisfaction: job placement services 


SATLAB 


Satisfaction: skill labs (writing, math, etc. 


SATSTORG 


Satisfaction: student organizations 


SATTRCRD 


Satisfaction: transfer credit assistance 


SATTUTOR 


Satisfaction: peer or other tutoring 


ACCHALL_STD 


Standardized academic challenge benchmark score 


ACTCOLL_STD 


Standardized active and collaborative benchmark 
score 


STUEFF_STD 


Standardized student effort benchmark score 


STUFAC_STD 


Standardized student-faculty interaction 
benchmark score 


SUPPORT_STD 


Standardized support for learners benchmark score 



After several multiple regressions five of the above variables were 
determined to be statistically significant predictors of "Entire 
Experience" responses. The following table summarizes the significant 
predictors of "Entire Experience" and their relative importance ranking. 



Ranking of the Impact of Each Significant Independent Variable on 
“Entire Experience, Holding All Other Variables Constant 



Rank Variable 

1 Quality relationships with faculty (ENVFAC) 

2 Support for learners (SUPPORT_STD) 

3 Grade point average (GPA) 

4 Satisfaction in computer labs" (SATCOMB) 

5 Academic challenge (ACCHALL_STD) 



"Quality relationships with faculty" has the most important impact on 
students' entire experience responses. The key role of faculty for 
creating a positive experience at NIACC is highlighted by this finding. 
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The next most significant effect on "Entire Experience" is "support for 
learners" as measured by the standardized support for learners 
benchmark scored This important finding regarding "support for 
learners" is enhanced by research findings that it is consistently 
correlated with measures of persistence (McClenney and Marti, 
2006:7). As persistence is the key to graduation we can't help but be 
encouraged that students' overall assessment of their experience at 
NIACC is linked to their favorable evaluation of NIACC support services 
for learners. 

The remaining three predictors of "Entire Experience" - "grade point 
average", "satisfaction in computer labs" and "academic challenge 
benchmark^" - share a direct relationship and approximately the same 
level of impact on "Entire Experience." 



^ See Appendix A for a description of the survey questions that comprise the 'support 
for learners' and 'academic challenge benchmark scores'.) 
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I NTRODUCTION 



Successful organizations, both public and private, rely heavily on 
"word-of-mouth recommendations" for their products and services. 
Financial viability of any organization depends profoundly on formal 
and informal networks of customers who pass their assessments of an 
organization's performance to acquaintances, friends and families. 

As "enrollment management" becomes more important for the fiscal 
heath of any higher education institution it is important to understand 
the dynamics and importance of 'word-of-mouth recommendations' 
passed on by existing students to their friends and family members. 
What determines students' positive or negative recommendations? 

One source of information to help answer this question is the 
Community College Survey of Student Engagement (CCSSE). This 
analysis explores the relationship between students' responses to the 
CCSSE question, "Would you recommend this college to a friend or 
family member?," and other survey questions dealing with the quality 
of relationships with other students, faculty and administrative 
personnel. In addition, the analysis examines the relationship between 
student recommendations and how they evaluate their entire 
educational experience at NIACC. In the final stage of the analysis the 
"entire experience" response was transformed from an explanatory 
variable utilized to understand student recommendations to a 
dependent variable to be explained by student responses to other 
CCSSE questions. 

Data 

The data for this analysis is from the CCSSE 2006 random sampling of 
592 students enrolled at NIACC. The following is a descriptive 
summary of the sample: 

■ 84% began their postsecondary education at NIACC 

■ 87% are full-time students 

■ Credits earned at time of survey: 

None 5.9% 

1-14 25.6% 

15-29 27.8% 

30-44 14.2% 

45-60 20.1% 

Over 60 6.3% 

■ 48.8% are male; 51.2% are female 
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■ 10.6% are married 

■ Highest academic credential at time of survey: 



None 


3.05% 


High School Diploma 


85.79% 


Vocational Technical Certificate 


5.08% 


Associate Degree 


4.74% 


Bachelor's Degree 


1.18% 


Masters/Doctorate/Professional 


0.17% 



■ 59.3% of students earned 0-29 credits 

■ 40.7% of students earned 30 or more credits 

■ 85.3% are traditional age students (24 and younger) 

■ 14.7% are 25 and older 

Variables i n the Analysi s 

Table 1 provides a summary of the variables in stage 1 of the analysis. 



Table 1 Variables In Stage 1 of the Analysis 


Name 


CCSSE Question/Instruction 


Coding 


RECOMMEND 

"Recommendation" 


Would you recommend this college to a 
friend or family member? 


1 = Yes 
0 = No 


ENTIREXP 
"Entire Experience" 


How would you evaluate your entire 
experience at this college? 


l=Poor 

2=Fair 

3=Good 

4=Excellent 


I Variables dealing with the quality of relationships with people: I 


ENVSTU 

"Environment Students" 


Mark the box that best represents the 
quality of your relationship with people 
at this college. 

Your relationship with: Other students 


Responses range from 1 to 7, 
with scale anchors described as 
(1) Unfriendly, unsupportive, 
sense of alienation 
(7) Friendly, supportive sense 
of belonging 


ENVFAC 

"Environment Faculty" 


Your relationship with: Instructors 


Responses range from 1 to 7, 
with scale anchors described as 
(1) Unfriendly, unsupportive, 
sense of alienation 
(7) Friendly, supportive sense 
of belonging 


ENVADM 

"Environment Admin" 


Your relationship with: Administrative 
personnel and offices 


Responses range from 1 to 7, 
with scale anchors described as 
(1) Unfriendly, unsupportive, 
sense of alienation 
(7) Friendly, supportive sense 
of belonging 
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Research Hypotheses 

This analysis tests the following 'stage 1' hypotheses: 



Table 2: Research Hypotheses: Stage 1 of the Analysis 


1. Recommendation as the dependent variable 


A. 


Basic Model 




Ho - RECOMMEND is not related to student ENVSTU, ENVFAC and ENVADM responses 

Hi - RECOMMEND is directly related to student ENVSTU, ENVFAC and ENVADM 
responses 


B. 


Expanded Model 




Ho - recommend is not related to student ENVSTU, ENVFAC, ENVADM and ENTIREXP 
responses 

Hi - RECOMMEND is directly related to student ENVSTU, ENVFAC, ENVADM and 
ENTIREXP responses 



Stati sti cal Tool 

Logistic Regression. As the dependent variable, RECOMMENDATION, 
is a binary categorical variable and because we are interested in 
controlling for an important variable that may impact the odds of 
making a positive recommendation the analytical tool of choice is 
logistic regression. Logistic regression is "the most important model for 
categorical response data" (Agresti, 2002:165). 

Logistic regression applies maximum likelihood estimation after 
transforming the dependent into a logit variable (the natural log of the 
odds of the dependent variable, positive recommendation, occurring or 
not). In this way, logistic regression estimates the probability of a 
positive recommendation occurring, controlling for other important 
predictor variables. 

Analysi s 

The first logistic model to be tested regresses "Recommendation" on 
the three quality relationship variables: "Environment Admin", 
"Environment Faculty" and "Environment Students." The regression is 
specified in the following form, Eq.l: 

Eq. 1 : Pr ob{RECOMMENDATION = 1 1 (x)) = X{a + /3,ENVADM + p^ENVEAC + B.ENVSTU ) 



where ?^( ) is the logit function, exp(x)/(l + exp(x)). The effects can be 
simply stated as the odds ratio. 
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Before we test this model we should perform a preliminary test on the 
three quality relationship variables - ENVSTU, ENVFAC and ENVADM. As 
these three predictors of "Recommendation" are ordinal variables an 
additional test is needed to determine if treating them as quantitative 
variable is permissible (Agresti, 2002). Such a test involves a two step 
process. First we run a more complex logistics regression having a 
separate parameter for each category of relationship. Recall responses 
range from 1 to 7, with scale anchors described as (1) Unfriendly, 
unsupportive, and a sense of alienation to (7) Friendly, supportive, 
and a sense of belonging. 

This more complex model, let's call it "Model 2" in this context, is then 
compared to a simpler model (Model 1) where the quality relationship 
variable is treated as a single quantitative variable. The comparison 
tests a Chi-square difference with df equal to the degrees of freedom 
associated with the complex model minus the degrees of freedom of 
the simpler model. 

Logistic regressions for all three "quality relationship" variables were 
run for both the simple and more complex models. Summary statistics 
needed to compute the Chi-square difference test are provided in 
Table 3. This statistic tests that the simpler model is adequate, given 
that model holds. In each case we conclude that treating the ordinal 
variable as a quantitative variable is permissible. Agresti indicates 
that, "It is advantageous to treat ordinal predictors in a quantitative 
manner when such models fit well" (Agresti, 2002: 191). 



Table 3 Testing Permissiveness of Treating the 
Ordinal Quality Variables as Quantitative Variables 


Likelihood Ratio 
Model 1 


df 

Model 1 


Likelihood Ratio 
Model 2 


df 

Model 2 


Difference 


df 


el 


ENVADM 




39.663 


1 


47.982 


6J_ 


8.319 


5 


0.140 


ENVFAC 




29.069 


1 


32.189 


6|_ 


3.12 


5 


0.681 


ENVSTU 




29.232 


1 


35.531 


6 


6.299 


5 


0.278 



Now that we have completed that preliminary test we can proceed to 
estimate the parameters of the logistics regression specified in Eq. 1. 



^ In contrast to traditional statistical procedures the researcher hopes not to reject Ho 
in this case 
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Table 4 reports the estimated parameters for the "Recommendation" 
logistics regression, Eq. 1: 



Table 4 Logistics Regression Parameter Estimates for Eo. 1 


Parameter 


Estimate 


Standard Error 


Z 


p-value 


95 % Confidence intervai 


Lower 


Upper 


1 CONSTANT 


-2.573 


0.704 


-3.655 


0.000 


-3.953 


-1.193 


2 ENVADM 


0.434 


0.112 


3.892 


0.000 


0.216 


0.653 


3 ENVFAC 


0.190 


0.092 


2.050 


0.049 


0.008 


0.372 


4 ENVSTU 


0.399 


0.121 


3.309 


0.001 


0.163 


0.636 


Likelihood Ratio = 57.361 with df = 3 and p = 0.000 
Naglekerke's R-square = 0.204 



The logit model (Eq. 1) is statistically significant. The reported 
likelihood-ratio (LR) tests that "Recommendation" is jointly 
independent of the predictors simultaneously; Hq: 3i = 32 = Ps = 0. 

The LR test statistic of 57.361 is chi-squared with 3 degrees of 
freedom and a p-value of 0.000. This demonstrates strong evidence 
that at least one predictor has an effect on Recommendation. 

Nagelkerke's R-square is an attempt to imitate the interpretation of 
multiple OLS R-square based on the likelihood. Nagelkerke's R-square 
can vary from 0 to 1. 

I NTERPRETATI ON OF COEFFI Cl ENTS 

Table 4 indicates that all three predictors of "Recommendation" are 
significant at the .05 level of confidence or better. Nevertheless, it is 
known that logistic coefficients may be found to be significant when 
the corresponding correlation is found to be not significant, and vice 
versa. To make certain statements about the significance of an 
independent variable, both the correlation and the logit should be 
significant. This additional test was completed. 

The predictor coefficients are large relative to their standard error and 
therefore appear to be an important predictor of "Recommendation." 
However, the interpretation of the logit coefficient is quite different 
from ordinary least squares The logit coefficient indicates how much 
the logit increases for a unit of change in the independent variable, but 
the probability of a 0 or 1 outcome is a nonlinear function of the logit. 
It is, therefore, more useful to turn to an evaluation of "odds ratio". 
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Odds Ratio I nterpretation 

Table 5 presents the odd ratios as well as standard errors and 
confidence intervals for the three predictors of "Recommendation." 



Table 5 Odds Ratio Estimates 


Parameter 


Odds Ratio 


Standard Error 


95 % Confidence intervai 


Lower 


Upper 


2 ENVADM 


1.544 


0.172 


1.241 


1.922 


3 ENVFAC 


1.209 


0.153 


0.944 


1.550 


4 ENVSTU 


1.491 


0.180 


1.177 


1.888 



The odds ratio table provides a more intuitive and meaningful 
understanding for the impact of the three predictors on 
"Recommendation." 

The odds ratio is a multiplicative factor by which the odds change 
when the independent variable increases by one unit, holding all other 
independent variables constant. The odds ratio for "Environment 
Admin" is 1.544^ 

A one unit increase in "Environment Admin", holding all other variables 
constant, improves the student's odds for providing a positive 
recommendation by 54%. We may say that when "Environment 
Admin" increases one unit, the odds that Recommendation = 1 (yes) 
increases by a factor of 54%, when all other variables are controlled. 

The odds ratio for "Environment Faculty" is 1.209. A one unit increase 
in "Environment Faculty", holding all other variables constant, 
improves the student's odds for providing a positive recommendation 
by 21%. We may say that when "Environment Faculty" increases one 
unit, the odds that Recommendation = 1 (yes) increases by a factor of 
21%, when all other variables are controlled. 

The odds ratio for "Environment Student" is 1.491. A one unit increase 
in "Environment Student", holding all other variables constant, 
improves the student's odds for providing a positive recommendation 
by 49%. We may say that when "Environment Student" increases one 
unit, the odds that Recommendation = 1 (yes) increases by a factor of 
49%, when all other variables are controlled. 



^ Given a logit coefficient, Pi, the odds ratio can be calculated exp(Pi). For example, 
the logit coefficient for "Environmental Admin" equals 0.434. The odds ratio equals 
exp(0.434) = 1.544. 
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Each odds ratio has a reported 95% confidence interval. For the 
variable "Environment Student" the lower boundary odds for a positive 
recommendation is nearly 18% with an upper boundary of 89%, 
holding all other variables constant. The reader is encouraged to 
inspect the confidence intervals for the other two predictors of 
"Recommendation." 

Given the known parameter estimates for the logistics regression, Eq. 

1 : 

Eq. 1 : Vrob{RECOMMENDATION = 1 1 (x)) = A.{a + jS^ENVADM + p^ENVEAC + B^ENVSTU) 

in Table 3 we can predict the logit for any given student with known or 
with assumed three predictor inputs. For example, what would be the 
logit and odds ratio for "Recommendation" for students with the 
following CCSSE responses to the environmental questions? 

For example: Student "A" responds to the following three quality 
questions in the following manner: 

Quality relationship with administration: Student response is 4 

Quality relationship with faculty: Student response is 5 

Quality relationship with students: Student response is 6 

Substituting the student's inputs into the estimated logit model (Eq.l 

and coefficients found in Table 3) produces this predicted logit: 

Eq. 1 : Pr ob{RECOMMENDATION = 1 1 (x)) = /l(« + p^ENVADM + (i^ENVEAC + B.ENVSTU ) 



= 2.573 + (.0.434)(4) + (0.190)(5) + (0.399 (6) = 7.653 

This student's probability for making a positive recommendation can 
be calculated using the following method: 

1. Calculate the odds ratio: 



exp(logit) = odds ratio = exp(7.653) = 2106.95 
2. Calculate probability: 



^ Odd Ratio 2106.957 

?xobability= = 

{\ + Odd Ratio) (1 + 2106.957) 



.999 
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The estimated positive recommendation probability for this student is 
.999. She has an estimated 99% probability of making a positive 
recommendation to family and friends, given her responses to the 
CCSSE survey. Checking the database we find that this student did 
respond favorably to the "recommendation" question. 

Expanded Model 



We now know that student responses to CCSSE environmental quality 
of relationship questions are highly predictive for making positive 
college recommendations. There are many other CCSSE survey 
questions that may add to our understanding and ability to predict 
student recommendations. In particular the CCSSE survey asks the 
question, "How would you evaluate your entire experience at this 
college?" If we add this response to the model will it improve our 
ability to understand and predict student recommendations? 



To answer this question we add ENTIREXP ("Entire Experience") 
variable to Eq. 1 to produce the following expanded logistics model: 



Eq. 2 : 



Vxob{RECOMMENDATION = 1 1 (x)) = /I 



'a + /3,ENVADM + p^ENVEAC + B.ENVSTU + 
^ B^ENTIREXP 



where A.( ) is the logit function, exp(x)/(l + exp(x:)). The effects can be 
simply stated as the odds ratio. 

Before we test this model we should perform a preliminary test on 
"Entire Experience". As "Entire Experience" is an ordinal variable an 
additional test is needed to determine if treating "Entire Experience" as 
a continuous variable is permissible. Such a test involves a two step 
process. First we run a more complex logistics regression having a 
separate parameter for each category of "Entire Experience" (Poor, 

Fair, Good and Excellent) minus 1. This logistics regression takes the 
following form: 

Eq. 3 : Yr ob[RECOMMENDATION = 1 1 (x)) = A[a + p,Poor + p^Eair + pfiood) 



where "Excellent" equals the reference category. This logistics 
regression produces a LR test statistic of 112.701 which is chi-squared 
with 3 degrees of freedom and a p value =0.000. 



This more complex model (Eq. 3) is then compared to a simpler model 
where "Entire Experience" was treated as a single quantitative 



14 




variable. Estimated parameters for this logistic regression are reported 
in Table 6: 



Table 6 Logistics Regression Parameter Estimates for Testing 
“Entire Experience” AS A Quantitative Variable 


Parameter 


Estimate 


Standard 

Error 


Z 


p-value 


95 % Confidence 
intervai 


Lower 


Upper 


1 CONSTANT 


-4.782 


0.766 


-6.243 


0.000 


-6.283 


-3.281 


2 ENTIREXP 


2.682 


0.318 


8.428 


0.000 


2.058 


3.306 


Likelihood-ratio (LR) = 1 10.80; df = 1; p = .000 
Naglekerke's R-square = 0.376 



A chi-squared difference test between the more complex and simpler 
model revealed a equal to 1.901 (112.701 - 110.80) with 2 df and 

p = 0.387. This statistic tests that the simpler model is adequate, 
given that model holds. This simplification (from an ordinal to a 
quantitative variable) seems permissible (p=0.387‘^). Agresti indicates 
that, "It is advantageous to treat ordinal predictors in a quantitative 
manner when such models fit well" (Agresti, 2002: 191). 

Having confirmed the legitimacy of treating the ordinal variable "Entire 
Experience" as a quantitative variable we proceed with estimating the 
parameters for Eq. 2. The estimated parameters for Eq. 2 are found 
below in Table 7. 



Table 7 Parameter Estimates for Logistics Model, Eq. 2 


Parameter 


Estimate 


Standard Error 


Z 


p-vaiue 


95 % Confidence intervai 


Lower 


Upper 


1 CONSTANT 


-6.621 


0.999 


-6.626 


0.000 


-8.580 


-4.663 


2 ENV_ADM 


0.246 


0.126 


1.958 


0.050 


0.000 


0.493 


3 ENV_FAC 


-0.175 


0.144 


-1.217 


0.224 


-0.458 


0.107 


4 ENV_STU 


0.430 


0.137 


3.148 


0.002 


0.162 


0.698 


5 ENTIREXP 


2.488 


0.353 


7.051 


0.000 


1.797 


3.180 


Likelihood Ratio = 125.702 with df = 4 and p = 0.000 
Naglekerke's R-square = 0.423 



The model is significant, p = 0.000. However, the coefficient for 
"Environment Faculty" is not significant (p = 0.224) while all the other 
predictors remain significant. As such, we must drop "Environment 
Faculty" from the model and recast Eq.2 to the following logistics 
model, Eq. 3: 



contrast to traditional statistical procedures the researcher hopes not to reject Ho 
in this case. 
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Eq. 3 : Pr ob{RECOMMENDATION = 1 1 (x)) = A{a + /3,ENVADM + B^ENVSTU + B^ENTIREXP) 

where ) is the logit function, exp(x)/(l + exp(x)). The effects can be 
simply stated as the odds ratio. 

This revised model produced the following parameter estimates, 
depicted in Table 8 and odds ratios in Table 9. 



Table 8 Parameter Estimates for logistic regression eo. 3 


Parameter 


Estimate 


Standard Error 


Z 


p-value 


95 % Confidence intervai 


Lower 


Upper 


1 CONSTANT 


-6.734 


0.995 


-6.770 


0.000 


-8.684 


-4.784 


2 ENV_ADM 


0.208 


0.122 


1.700 


0.089 


-0.032 


0.447 


3 ENV_STU 


0.373 


0.129 


2.897 


0.004 


0.121 


0.626 


4 ENTIREXP 


2.370 


0.336 


7.054 


0.000 


1.711 


3.028 


Likelihood Ratio = 124.201 with df = 3 and p = 0.000 
Naglekerke's R-square = 0.418 




Table 8 reveals that the revised model is statistically significant (p = 
0.000) but "Environment Admin" is now beyond our alpha of .05. As 
such, we must drop "Environment Admin" and recast another more 
parsimonious model without "Environment Admin", Eq. 4: 

Eq. 4 : Pr ob{RECOMMENDATION = 1 1 (x)) = A{a + B.ENVSTU + B^ENTIREXP) 

where ) is the logit function, exp(x)/(l + exp(x:)). The effects can be 

simply stated as the odds ratio. Table 10 reveals the estimated 
parameters associated with Eq. 4. 



Table 10 Parameter Estimates for Logistics Regression Eo. 4 




Parameter 


Estimate 


Standard Error 


Z 


p-vaiue 


95 % Confidence intervai 


Lower 


Upper 


1 CONSTANT 


-6.498 


0.973 


-6.677 


0.000 


-8.405 


-4.591 


2 ENV_STU 


0.413 


0.126 


3.283 


0.001 


0.166 


0.659 


3 ENTIREXP 


2.550 


0.323 


7.885 


0.000 


1.916 


3.184 


Likelihood Ratio = 121.312 with df = 2 and p = 0.000 
Hosmer-Lemeshow = 4.168 with 8 df and p = 0.842 
Naglekerke's R-square = 0.409 
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Table 11 reports the corresponding odds ratios. 



Table 1 1 Odds Ratio Estimates for Logistics Regression Eq. 4 


Parameter 


Odds Ratio 


Standard Error 


95 % Confidence intervai | 


Lower 


Upper 


2 ENV_STU 


1.511 


0.190 


1.181 


1.933 


3 ENTIREXP 


12.806 


4.141 


6.794 


24.137 



This model is statistically significant as well as all the predictors! We 
note that Naglekerke's R-square = 0.409, which is regarded as very 
satisfactory in the literature (Hensher and Johnson, 1981). In addition, 
the Hosmer-Lemeshow statistic (4.17 with p = 0.842) indicates a good 
fit of the model to the data. 

The odds ratio table reveals that a student's "Entire Experience" at 
NIACC has a tremendous impact on his/her odds of making a positive 
recommendation for the college to family and friends. An increase in 
one unit on the Entire Experience scale increases the odds of making a 
positive recommendation by nearly 13 times, holding constant 
"Environment Student." 

Students' quality relationships with other students still demonstrate 
significant effects on "Recommendation". Holding constant "Entire 
Experience", a one unit increase in student responses to "Environment 
Student" increases the odds for a positive recommendation by 51%. 
Clearly both variables are important but students' total experiences at 
NIACC overwhelm all other variables in this analysis. 

Given one of the primary purposes of logistic regression is to generate 
an equation that can reliably classify observations into one or two 
outcomes (positive recommendation versus a negative 
recommendation) we can check the model's predictive ability through 
a graphical means, the ROC (Receiver Operating Characteristic) curve. 
The ROC curve is presented below. 

The larger the area below the curve the better the model; that is, the 
better the predictions (Agresti, 2002). The area under the ROC curve 
is 0.885, which is identical to another measure of predictive power, the 
concordance index, c. The concordance index estimates the probability 
that the predictions and outcomes are concordant. A value of 0.5 
means predictions are no better than random guessing. A concordance 
index equal to .885 is considered quite satisfactory. 
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Receiver Operating Characteristic Curve 




1 - Specificity 



Accounting for "Entire Experience" Responses 

Now that we know that NIACC students' entire experiences 
significantly and in a huge manner impact their recommendation for a 
friend or family member to enroll at NIACC the next logical question is 
what determines students' responses to "Entire Experience." What 
variables are most highly and significantly related to how a student 
judges their entire experience at NIACC? 

Student responses on the CCSSE survey provide insights into many 
variables that could possibly impact students' responses to the "Entire 
Experience" response. Among the variables that appear to be of 
immediate interest are summarized in Table 12. 
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Table 12 Variables for the 
“Entire Experience” Anal ysis 


Name 


Description 


ENVSTU 


Quality of relationships with other students 


ENVFAC 


Quality of relationships with faculty 


ENVADM 


Quality of relationships with Administrative 
personnel and offices 


GPA 


College GPA 


SATACAD 


Satisfaction: academic advising/ planning 


SATCACOU 


Satisfaction: career counseling 


SATCHLD 


Satisfaction: child care 


SATCOMLB 


Satisfaction: computer lab 


SATDISAB 


Satisfaction: disability services 


SATFAADV 


Satisfaction: financial aid advising 


SATJOBPL 


Satisfaction: job placement services 


SATLAB 


Satisfaction: skill labs (writing, math, etc. 


SATSTORG 


Satisfaction: student organizations 


SATTRCRD 


Satisfaction: transfer credit assistance 


SATTUTOR 


Satisfaction: peer or other tutoring 


ACCHALL_STD 


Standardized academic challenge benchmark score 


ACTCOLL_STD 


Standardized active and collaborative benchmark 
score 


STUEFF_STD 


Standardized student effort benchmark score 


STUFAC_STD 


Standardized student-faculty Interaction 
benchmark score 


SUPPORT_STD 


Standardized support for learners benchmark score 


The "satisfaction" measures were coded 0=NA, l = Not at all, 2=Somewhat and 3=Very. Responses 1, 


2 and 3 can be treated as ordinal responses. "0" responses were treated as missing data. 



In order to determine if these variables significantly account for the 
variance in "Entire Experience" OLS (Ordinary Least Squares 
regression) was utilized as the statistical tool of choice. The specified 
OLS model for accounting for the variance in "Entire Experience" is 
found in Eq. 5. 

Eq. 5 : ENTIREXP = a + 5,SATACAD + fi^SATCACOU + fijSATCHLD + fi^SATCOMLB + 
fijSATDISAB + 5,SATFAADV + fi^SATJOBPL + 5gSATLAB + 
SgSATSTORG + fimSATTRCRD + fii,SATTUTOR + fii2ACCHALL_STD + 
fii3ACTCOLL_STD + Rj^STUEFF STD + fi^STUFAC STD + 
fii,SUPPORT_STD + B„GPA + B^^ENVSTU + B^^ENVFAC + B^^ENVADM +e 



The OLS regression of "Entire Experience" on the specified 
independent variables produced the results reported in Table 13. 
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Table 13 Regression Coefficients, Eq. 5 


Effect 


Coefficient 


Standard 

Error 


Std. 

Coefficient 


Toierance 


t 


P- 

vaiue 


ACCHALL_STD 


0.003 


0.001 


0.108 


0.637 


2.652 


0.008 


ACTCOLL_STD 


-0.001 


0.001 


-0.033 


0.537 


-0.733 


0.464 


ENV_ADM 


0.029 


0.020 


0.066 


0.512 


1.444 


0.149 


ENV_FAC 


0.139 


0.023 


0.266 


0.555 


6.086 


0.000 


ENV_STU 


0.016 


0.022 


0.031 


0.641 


0.751 


0.453 


GPA 


0.068 


0.018 


0.133 


0.870 


3.791 


0.000 


SATACAD 


0.077 


0.047 


0.077 


0.480 


1.642 


0.101 


SATACOU 


-0.063 


0.051 


-0.064 


0.405 


-1 .240 


0.216 


SATCHLD 


0.031 


0.043 


0.035 


0.442 


0.723 


0.470 


SATCOMLB 


0.090 


0.045 


0.077 


0.729 


2.022 


0.044 


SATDISAB 


-0.009 


0.042 


-0.009 


0.524 


-0.210 


0.834 


SATFAADV 


0.039 


0.039 


0.042 


0.598 


0.996 


0.319 


SATJOBPL 


0.039 


0.046 


0.039 


0.476 


0.828 


0.408 


SATLAB 


0.020 


0.044 


0.020 


0.596 


0.465 


0.642 


SATSTORG 


-0.041 


0.041 


-0.041 


0.617 


-1.000 


0.318 


SATTRCRD 


0.077 


0.043 


0.078 


0.560 


1.800 


0.072 


SATTUTOR 


0.019 


0.039 


0.020 


0.580 


0.470 


0.639 


STUEFF_STD 


0.000 


0.001 


0.012 


0.549 


0.280 


0.780 


STUFAC_STD 


0.000 


0.001 


-0.002 


0.468 


-0.032 


0.975 


SUPPORT_STD 


0.006 


0.001 


0.185 


0.440 


3.760 


0.000 


Multiple R = 0.627; Squared Multiple R = 0.393 
F-ratio = 18.480; p = 0.000 



It's clear that while the entire model is significant (p =0.000) a 
number of predictors of "Entire Experience" are not significant at the 
.05 level of confidence. Only five predictors are significant: 



ACCHALL_STD 


Standardized academic challenge benchmark score 


ENVFAC 


Quality of relationships with faculty 


GPA 


College GPA 


SUPPORT_STD 


Standardized support for learners benchmark score 


SATCOMLB 


Satisfaction: computer lab 



It is noted that satisfaction with transfer credit assistance approaches 
significance (p = 0.072) but doesn't make the cut point at .05. 

Given the above results the non-significant variables were deleted 
from the original "Entire Experience" model and a revised model was 
specified with the statistically significant variables. The revised model 
is specified in Eq. 6: 

Eq. 6 : ENTIREXP = a + fijENYFAC + fi^GFA + fijSATCOMLB + fi^ACCHALL STD + 
fijSUPPORTSTD + e 
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This model produced the following regression coefficients in Table 14 
and the associated Analysis of Variance in Table 15. 



Table 14 Regression Coefficients, Eq. 5 


Effect 


Coefficient 


Standard Error 


Std. 

Coefficient 


Toierance 


t 


p-vaiue 


ACCHALL_STD 


0.003 


0.001 


0.100 


0.846 


2.788 


0.005 


ENVFAC 


0.166 


0.019 


0.318 


0.788 


8.575 


0.000 


GPA 


0.065 


0.017 


0.128 


0.944 


3.778 


0.000 


SUPPORT_STD 


0.134 


0.040 


0.115 


0.932 


3.361 


0.001 


SATCOMLB 


0.008 


0.001 


0.267 


0.754 


7.061 


0.000 


Multiple R = 0.605; Squared Multiple R = 0.366 

‘Condition indices’ for the independent variables range between 1 .0 and 1 .88, providing confidence that 
multicollinearity is not a problem (Beisley, Kuh and Welsh, 1980) 







Table 15 Analysis of Variance 
“Entire Experience” 




Source 


ss 


df 


Mean Squares 


F- ratio 


p-vaiue 


Regression 


94.921 


5 


18.984 


67.552 


0.000 


Residuai 


164.683 


586 


0.281 







The revised "Entire Experience" model (Eq.5) is significant with an F- 
ratio of 67.55 and a p = 0.000. In addition, each predictor of "Entire 
Experience" is statistically significant at p = 0.005 or better. The 
model explains 36.6% of the variance in "Entire Experience." This is a 
satisfactory level of explained variance in educational research. 

The standardized coefficients (beta weights) in Table 14 provide a 
means to assess the relative impacts of the predictors of "Entire 
Experience". Standardized coefficients have the advantage of being 
scale-free indices and therefore can be compared across different 
variables^ (Pedhazur, 1982: 247). The beta weights indicate the 
expected change in "Entire Experience", expressed in standard scores, 
associated with a one standard deviation change in an independent 
variable, holding the remaining variables constant. 

Table 16 provides a ranking summary of the impact of each 
independent variable on "Entire Experience, holding all other variables 
constant. 



^The disadvantage of using beta weights is that they are sample-specific and can not 
be used for the purpose of generalization across settings and populations (Pedhazur, 
1982: 247). This is not an issue for this analysis as the research purpose is to 
understand the dynamics of "Recommendation" and "Entire Experience" and apply 
that knowledge to advance the mission of NIACC. 
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Table 16 Ranking of the Impact of Each Independent Variable on 
“Entire Experience, Holding All Other Variables Constant 



Rank Variable 

1 Quality relationships with faculty (ENVFAC) 

2 Support for learners (SUPPORT_STD) 

3 Grade point average (GPA) 

4 Satisfaction in computer labs" (SATCOMB) 

5 Academic challenge (ACCHALL_STD) 



Quality relationships with faculty (ENVFAC) has the most important 
impact on students' entire experience response. The key role of faculty 
for creating a positive experience at NIACC is highlighted by this 
finding. 

The next most significant effect on "Entire Experience" is support for 
learners (SUPPORT_STD), as measured by the standardized support 
for learners benchmark score. (See Appendix A for a description of the 
survey questions that comprise this benchmark.) 

This important finding regarding support for learners is enhanced by 
research findings that "support for learners" is consistently correlated 
with measures of persistence (McClenney and Marti, 2006:7). As 
persistence is the key to graduation we can't help but be encouraged 
that students' overall assessment of their experience at NIACC is 
linked to their favorable evaluation of NIACC support services for 
learners. 

The remaining three predictors of "Entire Experience" are nearly equal 
in impact. GPA and "Entire Experience" are directly related. A one unit 
increase in GPA standard deviation is associated with a .13 standard 
deviation in "Entire Experience." A similar relationship exists between 
"satisfaction in computer labs" (SATCOMB) and "Entire Experience" 
with a beta weight equal to .11. Finally, "academic challenge 
benchmark" (ACCFIALL_STD) shares a direct relationship with "Entire 
Experience" at about the same level of impact (.10). 

Conclusion and Discussion 

Recommendations that existing NIACC students pass on to their 
acquaintances, friends and families to attend NIACC are vital to 
NIACC's important mission and its fiscal health. Quality of student 
relationships with people, specifically administrative personnel, faculty 
and other students have been found to be significant predictors of 
student recommendations to attend NIACC. 



22 





Stage One Findings. In the first stage of the analysis logistics 
regression was utilized to assess the relative importance of variables 
dealing with the quality of student relationships with people, 
specifically administrative personnel, faculty and other students. Major 
findings for this first stage of the analysis include: 

■ Positive experiences (measured by a one unit change in student 
response) with administrative personnel, holding all other 
variables constant, improves the student's odds for providing a 
positive recommendation by 54%. 

■ Positive experiences with other students, holding all other 
variables constant, improve the student's odds for providing a 
positive recommendation by 49%. 

■ Positive experiences with faculty, holding all other variables 
constant, improve the student's odds for providing a positive 
recommendation by 21%. 

There are many other CCSSE survey questions that add to our 
understanding and ability to predict student recommendations. In 
particular the CCSSE survey asks the question, "How would you 
evaluate your entire experience at this college?" If we add this 
response to the model will it improve our ability to understand and 
predict student recommendations? 

Major findings associated with adding the "Entire Experience" response 
to the model include the following: 

■ Positive experiences with faculty and administrative personnel no 
longer are statistically significant and were therefore dropped 
from the explanatory model. 

■ Students' entire experiences at NIACC have a tremendous 
impact on their odds of making a positive recommendation for 
the college to family and friends. 

> An increase in one unit on the Entire Experience scale 
increases the odds of making a positive recommendation 
by nearly 13 times, holding the environmental variable, 
student relationships with each other, constant. 

■ Quality relationships with other students still demonstrate 
significant effects on "Recommendation". A one unit increase in 
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student responses to relationships with other students increases 
the odds for a positive recommendation by 51%. 

■ Clearly both variables - students' "entire experiences" at NIACC 
and their "relationship with each other" - are important but 
students' total experiences at NIACC overwhelm all other 
variables in this analysis. 

Stage Two Findings. As students' entire experiences are so critical for 
determining their recommendations to attend NIACC a second stage 
analysis was undertaken to determine factors that account for the 
variance in the "Entire Experience" variable. In other words, "Entire 
Experience" was moved from an independent variable to a dependent 
variable to be explained. 

Student responses on the CCSSE survey provide insights into many 
variables that could possibly impact students' responses to the "Entire 
Experience" question. Among the variables that were of immediate 
interest included the following: 



Variables for the 
“Entire Experience” Anal ysis 


Name 


Description 


ENVSTU 


Quality of relationships with other students 


ENVFAC 


Quality of relationships with faculty 


ENVADM 


Quality of relationships with Administrative 
personnel and offices 


GPA 


College GPA 


SATACAD 


Satisfaction 


academic advising/ planning 


SATCACOU 


Satisfaction 


career counseling 


SATCHLD 


Satisfaction 


child care 


SATCOMLB 


Satisfaction 


computer lab 


SATDISAB 


Satisfaction 


disability services 


SATFAADV 


Satisfaction 


financial aid advising 


SATJOBPL 


Satisfaction 


job placement services 


SATLAB 


Satisfaction 


skill labs (writing, math, etc. 


SATSTORG 


Satisfaction 


student organizations 


SATTRCRD 


Satisfaction 


transfer credit assistance 


SATTUTOR 


Satisfaction 


peer or other tutoring 


ACCHALL_STD 


Standardized academic challenge benchmark score 


ACTCOLL_STD 


Standardized active and collaborative benchmark 
score 


STUEFF_STD 


Standardized student effort benchmark score 


STUFAC_STD 


Standardized student-faculty interaction 
benchmark score 


SUPPORT_STD 


Standardized support for learners benchmark score 
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After several multiple regressions five of the above variables were 
determined to be statistically significant predictors of "Entire 
Experience" responses. The following table summarizes the significant 
predictors of "Entire Experience" and their relative importance ranking. 



Ranking of the Impact of Each Significant Independent Variable on 
“Entire Experience, Hoiding Aii Other Variabies Constant 



Rank Variable 

1 Quality relationships with faculty (ENVFAC) 

2 Support for learners (SUPPORT_STD) 

3 Grade point average (GPA) 

4 Satisfaction in computer labs" (SATCOMB) 

5 Academic challenge (ACCHALL_STD) 



"Quality relationships with faculty" has the most important impact on 
students' entire experience responses. The key role of faculty for 
creating a positive experience at NIACC is highlighted by this finding. 

The next most significant effect on "Entire Experience" is "support for 
learners" as measured by the standardized support for learners' 
benchmark scored This important finding regarding "support for 
learners" is enhanced by research findings that it is consistently 
correlated with measures of persistence (McClenney and Marti, 
2006:7). As persistence is the key to graduation we can't help but be 
encouraged that students' overall assessment of their experience at 
NIACC is linked to their favorable evaluation of NIACC support services 
for learners. 

The remaining three predictors of "Entire Experience" - "grade point 
average", "satisfaction in computer labs" and "academic challenge 
benchmark®" - share a direct relationship and approximately the same 
level of impact on "Entire Experience." 

The importances of these findings now need to be examined within a 
context of continuous quality improvement. Specifically, how can 
NIACC maximize the positive findings to optimize enrollments? 

Second, why aren't some of the CCSSE variables significantly related 
to student recommendations or to their entire experience responses? 



® See Appendix A for a description of the survey questions that comprise the 'support 
for learners' and 'academic challenge benchmark scores'.) 
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Appendix A 
CCSSE Benchmarks 



Academic Challenge Benchmark: 

Benchmark composed often survey items. A four-item response scale 
(Never, Sometimes, Often, Very often) is used for the following 
Academic Challenge related college activity: 

■ Worked harder than you thought you could to meet an 
instructor's standards or expectations 

A four-item response scale (Very little. Some, Quite a bit. Very much) 
is used for the following mental activity items: 

■ Analyzing the basic elements of an idea, experience, or theory 

■ Synthesizing and organizing ideas, information, or experiences in 
new ways 

■ Making judgments about the value or soundness of information, 
arguments, or methods 

■ Applying theories or concepts to practical problems or in new 
situations 

■ Using information you have read or heard to perform a new skill 
A five-item response scale (None, Between 1 and 4, Between 5 and 
10, Between 11 and 20, More than 20) is used for the following 
academic preparation items: 

■ Number of assigned textbooks, manuals, books, or book-length 
packs of course readings 

■ Number of written papers or reports of any length 

A seven-item response scale (Ranging from 1 to 7, with scale anchors 
described: (1) Extremely easy (7) Extremely challenging) is used for 
the following exam item: 

■ Mark the box that best represents the extent to which your 
examinations during the current school year have challenged you 
to do your best work at this college 

A four-item response scale (Very little. Some, Quite a bit. Very much) 
is used for the following college opinion item: 

■ Encouraging you to spend significant amounts of time studying 

Support for Learners Benchmark 

This support benchmark is composed of seven survey items. A four 
item response scale (Very little. Some, Quite a bit. Very much) was 
used by CCSSE to construct the benchmark from the following college 
survey questions: 
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■ Providing the support you need to help you succeed at this 
college 

■ Encouraging contact among students from different economic, 
social, and racial or ethnic backgrounds 

■ Helping you cope with your non-academic responsibilities (work, 
family, etc.) 

■ Providing the support you need to thrive socially 

■ Providing the financial support you need to afford your education 

In addition, a four-item response scale (Don't know/N.A., Rarely/ 
never, Sometimes, Often) was used for the following student services 
items: 

■ Frequency: Academic advising/planning 

■ Frequency: Career counseling 
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